Don’t Expect Too MucuH: THE LIMITED 
NEPC USEFULNESS OF COMMON SES MEASURES 
—-—- AND A PRESCRIPTION FOR CHANGE 


POLICY CENTER 


Michael Harwell 
University of Minnesota 


January 2018 


National Education Policy Center 


School of Education, University of Colorado Boulder 
Boulder, CO 80309-0249 
(802) 383-0058 
nepc.colorado.edu 


Kevin Welner 
Project Director 


William Mathis 


Managing Director 


Don Weitzman 
Academic Editor 


Alex Molnar 
Publications Director 


Publication Type: NEPC Policy Briefs synthesize existing research knowledge on a policy or practice 
issue of importance. NEPC Policy Briefs are blind peer-reviewed. 


Peer Review: This NEPC Policy Brief was blind peer-reviewed by members of the Editorial Board. Find 
NEPC publications at: http://nepc.colorado.edu/publications/all. Find the NEPC editorial board at: 
http://nepc.colorado.edu/editorial-board. 


Funding: This brief is made possible in part by funding from the Great Lakes Center for Education 
Research and Practice. 


Suggested Citation: Harwell, M. (2018). Don’t Expect Too Much: The Limited Usefulness of Common 
SES Measures and a Prescription for Change. Boulder, CO: National Education Policy Center. Retrieved 
[date] from http://nepc.colorado.edu/publication/SES. 


This material is provided free of cost to NEPC’s readers, who may make non-commercial use of 
thematerial as long as NEPC and its author(s) are credited as the source. For inquiries about commercial 
use, please contact NEPC at nepc@colorado.edu. 


NEPC 


NATIONAL EDUCATION 
POLICY CENTER 


Don’t Expect Too Mucu: THE LIMITED 
USEFULNESS OF COMMON SES MEASURES 
AND A PRESCRIPTION FOR CHANGE 


Michael Harwell 
University of Minnesota 


January 2018 


Executive Summary 


Measures of socioeconomic status (SES) are widely used in educational research and policy 
applications, in large part due to overwhelming evidence linking SES to student achieve- 
ment. SES is usually conceptualized as an unobservable factor—a construct—measured us- 
ing variables such as parental education, occupation, income/wealth, and home possessions 
to take into account disparities between students, classrooms, and schools. For example, 
assessing the efficacy of an intervention designed to improve students’ understanding of 
math, by comparing students who did not receive the intervention, would require adjusting 
for SES differences between the two groups of students. SES is taken into account statistical- 
ly to ensure that the program’s effectiveness is evaluated independent of student SES. This is 
because a failure to take SES disparities into account can result in concluding that an inter- 
vention is effective when much of its apparent effectiveness is related to the SES differences 
between students receiving and not receiving the intervention. 


A common way of controlling for SES uses students’ eligibility for a free- or reduced-price 
lunch (FRL). Yet researchers generally agree that FRL is a poor SES measure, since it fails to 
validly reflect access to financial resources, shows reduced levels of participation as students 
age, and misclassifies, on average, approximately 20% of students. 


Beyond problems with the continued use of the profoundly flawed student FRL variable, we 
face two additional problems with SES measures. First, the usefulness of common SES mea- 
sures, used at the student level, is undermined by the modest correlations that exist between 
those measures and outcome variables. This limits the effectiveness of SES measures as con- 
trol variables in statistical analyses. Second, common SES measures conceptualize SES in 
ways that are not guided by theory, thereby creating uncertainty about what SES represents 
(e.g., access to financial or social resources). Such atheoretical approaches to SES have a 
long history. But models grounded in thoughtful theory should be employed that define SES 
in ways consistent with the purpose of the research or policy, that guide the selection of SES 
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measures, and that help ensure that new SES measures do in fact measure what they are 
intended to (i.e., show evidence of construct validity). 


These problems speak to an important gap between the usefulness of common SES measures 
and the perception among researchers and policymakers of the great impact of SES in edu- 
cational settings. The problems, however, also provide a prescription for change, to promote 
a deeper understanding and more effective use of SES in research and policy. 


Recommendations include the following: 


e A theory-grounded model of SES should be adopted to define this construct in 
ways consistent with the purpose of the research or policy application. 


Correlations between SES measures and outcomes should be examined to assess 
the usefulness of these measures as control variables in statistical analyses. 


Researchers and policymakers wishing to employ existing SES measures should 
consider a composite index of SES, perhaps in conjunction with common mea- 
sures, or turn to alternative measures such as (a) students’ perception of their 
SES or (b) poverty estimates at the district level. Those interested in developing 
new measures should use a theory-grounded SES model as a guide to help ensure 
new SES measures do in fact measure what they are intended to (i.e., show evi- 
dence of construct validity). 


The development of new SES measures guided by a theory-grounded model of 
SES requires assembling a multidisciplinary team with expertise in a substantive 
area of education (e.g., mathematics education) as well as expertise in psycho- 
metrics, statistics, and the SES literature. 


Eligibility for a free- or reduced-price lunch should not be used as a student-level 
SES measure, but aggregating this variable to reflect the percentage of students 
receiving subsidized meals produces a crude but useful index to compare the eco- 
nomic need of a school or district with other schools or districts. 
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Introduction 


Socioeconomic status (SES) has a major presence in educational research. SES measures 
are used to increase statistical power and enhance causality arguments, and as control vari- 
ables in statistical models to take into account disparities among students, classrooms, and 
schools.t Socioeconomic measures also play a key role in allocating resources to schools 
through state and local education agencies.” The terms social class, social stratification, and 
SES are sometimes used interchangeably, and while there are some differences, they will 
be treated here as synonymous. SES also differs from poverty with the latter more narrowly 
defined because it is typically linked to the federal government’s poverty levels, which are 
defined using income. 


Multiple definitions of SES exist but most overlap with that appearing in a U.S. Department 
of Education (2012) report: “SES is defined as one’s access to financial, social, cultural, and 
human capital resources.” (p. 14) The widespread use of SES in education is in large part 
due to a belief that SES has a strong impact on many educational outcomes. Evidence of 
this belief is provided by the Coleman et al. (1966)4 report which included measures of SES 
such as the presence of reading materials in the home and parental education, the practice 
of reporting National Assessment of Educational Progress (NAEP) scores by student partic- 
ipation in the free/reduced price lunch (FRL) program, and the use of SES in theories of 
factors impacting educational attainment.®°” 


Other evidence of the perceived importance of SES is provided by a review of all papers 
published in two prominent educational research journals, American Educational Research 
Journal and Sociology of Education.’ Approximately 25% of the articles used achievement 
data as outcomes, and of these approximately 70% statistically controlled for the effects of 
SES measures. Repeating the survey for the same journals for 2000-2015 showed 29% of the 
articles used achievement data and approximately 62% controlled for measures of SES. Sim- 
ilarly, more than 50% of the articles in Educational Researcher from 2000-2015 modeling 
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educational outcomes employed SES measures in some capacity. In short, SES is a core facet 
of much educational research and policy because of the perceived importance of taking into 
account disparities in SES among students, classrooms, and schools to help ensure accurate 
inferences about student learning and achievement. For example, assessing the efficacy of 
an intervention designed to improve students’ understanding of math may require taking 
SES into account to evaluate whether its effectiveness is related to a student’s SES. 


To take disparities in SES into account, the U.S. Department of Education? recommends us- 
ing parental education, occupation, income or wealth, and home possessions (e.g., number 
of books, electronic tablets, and computers in a household) as SES measures, either indi- 
vidually or as a composite, because these variables are believed to reflect access to financial 
or social resources that many researchers and policymakers link to SES, and may have less 
measurement error than competing measures. The long history of these variables as SES 
measures in education and the consistent way variables like parental education have been 
measured (i.e., highest degree earned, total years of formal education) also contributes to 
recommendations to use these variables as SES measures. Another recommendation in the 
U.S. Department of Education (2012)"° report is to conceptualize SES as a construct (unob- 
servable variable or latent factor) that represents access to different kinds of resources, such 
as financial or social resources. 


It is important to distinguish between income and wealth, with the latter including accumu- 
lated economic assets such as property, housing, retirement balances, etc., and to recognize 
that changes in the distribution of wealth in the U.S. have been greater than changes in in- 
come in the past few decades." SES measures are assumed to provide similar information 


about access to similar resources, and correlations between measures and educational out- 
comes are assumed to be non-negligible. If those correlations are modest, the ability of SES 
measures to take disparities in SES into account in statistical analyses is limited. 


An important point of agreement in the SES literature is that a student’s FRL status (1 = el- 
igible, o = no), which generally involves students or parents returning an application, while 
relatively non-intrusive and cheap to obtain, is a poor measure of SES because it (a) fails to 
accurately reflect access to financial resources,” (b) shows reduced levels of participation as 
students age, and (c) misclassifies approximately 20% of students on average. Student FRL 
is often aggregated to a school or district level to produce the percentage of students eligible 
for a FRL, which is used in statistical analyses as well as to allocate resources in policy appli- 
cations primarily through Title I funds. School FRL values 40% or greater means a school is 
eligible for Title I funds which are used to help students from lower SES backgrounds meet 
academic standards by, for example, adding teachers, paraprofessionals, or instructional 
materials to a curriculum focused on these students. 


These difficulties produce an important gap between the perceived impact of SES in educa- 
tional settings and the usefulness of common SES measures that can bias research- and poli- 
cy-based conclusions. However, responding to these difficulties also provides a prescription 
for change that depends on defining this construct and selecting measures that are con- 
sistent with that definition and are correlated with important educational outcomes. The 
result should promote a deeper understanding and more effective use of SES in educational 
research and policy. 


http://nepc.colorado.edu/publication/SES 


Review of the Literature: SES and its 
Impact on Educational Outcomes 


Theory-grounded versus atheoretical approaches to SES 


Research places a premium on theory because theory defines phenomena of interest and 
their relationships, guides methodological decisions, and provides a framework for inter- 
preting empirical findings.’ Theory often motivates the selection of outcome variables in 
research studies but this practice is rarely followed for SES measures. Several authors have 
pointed out that atheoretical approaches to SES have a long history in multiple fields includ- 
ing education, psychology, and public health.* Lack of agreement on what SES represents 
and how it should be measured have exacerbated difficulties created by an atheoretical ap- 
proach. 


Educational studies often provide theoretical arguments justifying the selection of outcome 
variables but rarely do so for SES measures. An atheoretical approach to SES in education- 
al research is reflected in the undesirable practice of defining SES in research and policy 
applications by whatever measures of SES are used.’® For example, Morgan et al. (2015)? 
used a composite of parents’ education level, occupation, and income, along with student 
FRL status, in their study of the relationship between mathematics instructional practices 
and mathematics achievement. Similarly, Bachman et al. (2015)* used parental income and 
mother’s education as SES measures in their analyses of mathematics achievement data. 
Both studies provided theoretical arguments for outcome variables but neither defined SES 
(e.g., access to financial resources, access to social resources) or explained why particular 
measures were selected, creating uncertainty about what SES is and whether the selected 
SES measures are effective control variables for taking disparities in SES among students, 
classrooms, and schools into account. 


Atheoretical approaches can also create challenges in interpreting findings when SES is used 
to take disparities into account statistically (i.e., its effect is removed from other variables). 
Implicit in controlling for SES is understanding what is being controlled (removed); other- 
wise the findings may be difficult to correctly interpret, increasing the likelihood of biased 
conclusions. For example, statistically removing the effects of a pre-test of comprehension 
of science diagrams from a post-test whose construction was guided by a theoretical model 
clarifies what was removed (prior proficiency in comprehending diagrams) and enhances in- 
terpretations of findings.’? On the other hand, statistically removing the effects of SES mea- 
sures like student FRL status from an outcome reflecting mathematics achievement without 
ever defining SES or providing a rationale for utilizing FRL status makes it unclear what the 
removed variation represents. 


An atheoretical approach can also produce uncertainty about whether SES is unidimensional 
(one factor such as access to financial resources) or multidimensional (two or more factors 
such as access to financial and social resources). This distinction also impacts the measure- 
ment of SES. The U.S. Department of Education (2012)?° report distinguished between SES 
components -- variables that should be part of the measurement of SES -- and correlates, 
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-- variables that correlate with SES but are not part of SES and may not be suitable measures 
because what they are measuring may be unclear. For example, in an atheoretical approach 
it may be unclear whether a measure of a student’s access to individuals with advanced ed- 
ucational degrees is a component or correlate of SES. 


Concerns about atheoretical approaches to SES have been widely recognized: “All too often 
SES and class are ambiguous terms that serve as a shorthand expression to refer to social 
and economic characteristics that are thought to be important, but whose rationale or mean- 
ing is not always made clear” (Bollen et al., 2001, p. 4)", and “One of the most glaring gaps 
we discovered is between the theoretical literature on class or SES and the manner in which 
these concepts appear in empirical work.” (Bollen et al., 2001, p. 42)?" Unfortunately there 
is little evidence that this issue has received much attention from educational researchers 
and policymakers. 


Problems created by an atheoretical approach can be remedied by employing a theoretical 
model that conceptualizes SES in ways consistent with the definition and purpose of the 
particular research or policy application, guides the selection of SES measures, and plac- 
es a premium on construct validity in the construction of new measures (i.e., ensuring a 
new measure measures what it is intended to). Theoretical models and methodologies for 
measuring SES are available in several disciplines but all have roots in the social class lit- 
erature.*3 Prominent theoretical approaches to SES in this literature include those derived 
from Marx,” Weber,?5 Durkheim,”° and Coleman.”’ These theories are complex, often linked 
to different methodologies for measuring SES, and are not tied to substantive research areas 
like education. There are also a large number of competing theories. 


Coleman’s (1988)?° theory of social capital is among the most widely studied in education 
and is characterized by the critical role played by non-economic factors. Central to Cole- 
man’s theory is a convergence between economics and sociology combined in a model char- 
acterized by the actions of individual agents based on other actors and resources that are 
related through common interests and control.?? Social inequality results from differences 
in interests and control over scarce resources. The idea of resources is developed through 
material capital, human capital, and social capital, with SES representing access to these 
resources deriving from (a) material endowments such as income and wealth, (b) skills, 
abilities and knowledge reflected in educational level, and (c) an individual’s social network 
and the status, power, trustworthiness, and abilities of its members. For example, Coleman’s 
theory might posit families with different income and wealth levels (material capital) that 
become close through the friendship of their children (social capital), leads one set of par- 
ents to emphasize to their children the value of doing well in school based on their relation- 
ship with parents holding these values. 


Several authors have argued that a strength of Coleman’s theory is its ability to incorporate 
societal evolution through the impact of material, human, and social capital on the defini- 
tion and identification of social classes.2° These changes may also mediate the effect of SES 
on educational outcomes. For example, the continuing shift of income and wealth to the 
highest SES levels, the increasing numbers of women and ethnic-minority executives, newly 
minted high-tech millionaires and billionaires, the decline of manufacturing jobs, and the 
increased numbers of part-time employees may have an important impact on definitions of 
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SES and how this construct is measured. 


Adoption of a theoretical model of SES begins by conceptualizing SES as access to desired 
resources. For example, choosing Coleman’s theory to guide construction of a composite 
index of SES using multiple measures might involve providing evidence of the impact of ma- 
terial, human, and social capital factors on outcomes like students’ math and verbal achieve- 
ment. The choice between a composite index of SES, which implies SES is unidimensional, 
and multiple (multidimensional) factors not part of a composite, has been long debated.?" 
Assuming SES is unidimensional offers simpler explanations but may be unrealistic in some 
contexts because each factor making up SES (e.g., access to material resources, access to 
social resources) exerts an influence similar to other factors. Multidimensional SES may 
be more realistic but assumes that each factor can have an effect independent of others; 
this can lead to complicated explanations. There is agreement, however, that this decision 
should be informed by theory and the existing literature. 


Researchers should select SES measures based on their conceptual fidelity with SES factors 
as well as empirical evidence of construct validity.3? Other criteria that may figure in the se- 
lection of SES measures include measurement error, intrusiveness, cost, and the likelihood 
of missing data. These criteria have particularly important implications in practice. The 
popularity of FRL status as an SES measure is related to the fact it is relatively non-intru- 
sive, can be collected directly from a school, and is generally inexpensive to obtain. Other 
SES measures such as parental income, wealth, and educational level must typically be col- 
lected from parents, which may be perceived as intrusive and lead to missing data that bias 
findings. For example, higher-income households are less likely to provide income informa- 
tion. Gathering SES data on parental income, wealth, and educational level may also require 
greater resources than those needed to obtain student FRL status for several reasons. 


Researchers should select A researcher wishing to take disparities in SES 


ens measires based on then “mene students, classrooms, or schools into ac- 

: ; ; count may argue that parental education and home 

conceptual fidelity with SES possessions show conceptual fidelity if SES is de- 

factors as wellas empirical _ fined as access to material capital, and choose one 

evidence of construct validity or both of these measures, guided by evidence of 

construct validity, anticipated measurement error, 

likelihood of missing data, and cost. Evidence of the relationship between SES measures of 

a construct and educational outcomes may also contribute to this choice because stronger 

relationships should result in more effective efforts to take disparities in SES into account 
in statistical analyses. 


A particularly helpful way to proceed is to employ a (structural equation) diagram to illus- 
trate the SES model and its measures, and to use a regression-based statistical procedure 
(structural equation modeling) to provide empirical evidence that a variable is measuring 
the underlying SES factor (construct validity) or that SES has an impact on educational out- 
comes. Similar diagrams and analyses have been used in research.?3 In the Oakes and Rossi 
(2003)34 SES model, material capital, human capital, and social capital served as factors 
that were treated as unidimensional and posited to have a direct impact on a comprehensive 
factor labeled SES (access to desired resources). Oakes and Rossi assumed SES factors were 
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correlated. The diagram neatly defined SES and its measures, including parental income 
(material capital), parental education (human capital), and the number of clubs a house- 
hold’s adults participated in (social capital). These authors used data obtained from the Na- 
tional Survey of Families and Parents to form a composite index and estimate an SES score 
for each case based on their model. The resulting model provided evidence of construct va- 
lidity: higher SES scores reflected greater access to desired material, human, and social re- 
sources (e.g., respectively, home possessions such as access to a computer and the internet, 
occupations whose flexibility facilitates parental access to their child’s teachers and school 
activities, and access to individuals with advanced degrees who provide tutoring), and lower 
scores reflected less access. 


Consider a hypothetical example in educational research of adopting a theoretical model of 
SES similar to that of Oakes and Rossi (2003).°5 A researcher plans to assess the impact of 
student race on math achievement after an extensive effort to control for SES. The findings 
are expected to have important implications for efforts to eliminate the achievement gap: if 
the gap shrinks or disappears once disparities in SES are taken into account the inference 
is that Black-White achievement differences are partly or wholly attributable to differences 
in SES, whereas if the gap doesn’t change the inference is that Black-White achievement 
differences exist independent of SES. If Title I schools participated in the study either find- 
ing could impact the way Title I funds might be used to help students from lower SES back- 
grounds meet academic standards. 


Based on a literature review, the researcher defines SES as access to desired resources and 
turns to Coleman’s theory of social capital to define access as involving access to material, 


human, and social capital. The literature also prompts the researcher to treat these factors as 
multidimensional and to posit that access to material capital will have a direct impact on ac- 
cess to social capital, which in turn is expected to have a direct impact on math achievement. 


Figure 1: Hypothetical Model of the Impact of Multidimensional SES on Math Achievement 


http://nepc.colorado.edu/publication/SES 


Figure 1 provides a (structural equation) diagram of the above model for students. Race 
and access to material capital and human capital are constructs (unobserved factors, latent 
variables) serving as independent variables which are assumed to impact the dependent 
variable math achievement, access to social capital is both a latent independent and depen- 
dent variable, math achievement is a latent dependent variable, and X1-X13 are observed 
SES measures (error terms have been omitted). Student race is assumed to be measured 
with no error and is measured with a single observed variable X1. The remaining variables 
are treated as components and represent observed measures of a construct. For example X2- 
X4 could be parental income, wealth, and home possessions, X5-X7 could be the number of 
clubs a parent participated in, access to individuals with advanced degrees, and size of peer 
group, and X8-X10 could be parental education, occupation, and years in current job. 


Single-headed arrows in Figure 1 represent the direct impact (effect) of one variable on 
another and are consistent with traditional methods for estimating the impact of SES (e.g., 
effect of access to material capital on math achievement estimated using a correlation co- 
efficient, regression slope, or path coefficient). Figure 1 also contains indirect (mediated) 
effects in which a variable serves as both an independent and dependent variable. Examples 
of mediators characterizing the impact of SES on educational outcomes are not new®° and 
Figure 1 posits that the impact of access to material capital on math achievement is mediated 
by access to social capital (i.e., the impact of access to material capital on math achievement 
is transmitted through access to social capital). Perhaps the impact of greater income on 
math achievement is amplified in families with greater access to individuals with advanced 
degrees. As with direct effects, theory and previous literature should inform the use of me- 
diated effects like those in Figure 1. 


Representing the impact of SES through both direct and indirect effects, and the total effect 
which is the sum of direct and indirect effects, offers a potentially useful complement to tra- 
ditional approaches in providing a comprehensive picture of the impact of SES. 


Modest correlations between SES measures and outcome variables limit 
the effectiveness of SES as a control variable in statistical analyses 


If SES has a strong impact on educational outcomes, non-negligible correlations reflecting 
this impact should appear in the literature. Available evidence involving correlations sug- 
gests these correlations are on average moderately weak (.20[] r [] .30) to weak ([] .20) for 
student-level data but show a wide range of values. Moderately weak to weak correlations 
limit the effectiveness of SES measures as control variables in statistical analyses because a 
modest amount of variance in an outcome variable is removed. For example, a correlation 
of r = .22 between parental education and math achievement means (.22)? or about 5% of 
the variance in the latter has been removed (taken into account, controlled). In other words, 
removing the variation in math achievement attributable to parental education makes it 
possible to make inferences about students’ math achievement without worrying that these 
are distorted by parental education. Note that a correlation of r = 1.0 means all variation in 
math achievement is attributable to parental education and a correlation of r = 0 that SES 
has no effect on math achievement and there is no variation in achievement attributable to 


http://nepc.colorado.edu/publication/SES 11 of 21 


SES. 


White’s (1982)3” meta-analysis is probably the most widely cited source of the SES- achieve- 
ment relationship, with 386 citations between 1983 and 2016 found using a Web of Sci- 
ence search. White’s meta-analysis was based on correlations between SES and achievement 
measures obtained from 101 published and unpublished studies appearing between 1918 and 
1975. White (1982) reported an unweighted mean correlation of T = .24 based on averaging 
K = 489 correlation coefficients for student-level data. Sirin (2005)3° updated the results of 
White by coding the SES-achievement relationship in 58 journal articles appearing between 
1990 and 2000 and performing a meta-analysis. For K = 207 student-level correlation be- 
tween SES and achievement, and a statistical model that assumed studies producing these 
correlations were sampled from a population of studies with varying SES-achievement cor- 
relations, fT =.27 (T = .28 for a statistical model assuming the population of studies shared a 
common SES-achievement correlation). 


Harwell, Maeda, Lee, and Xie (2017)3° performed a meta-analysis of the relationship be- 
tween SES and achievement based on sampling 143 studies in which the most important 
findings were (a) T= .18 for K = 2,610 for student-level data when sample sizes in studies 
were ignored, (b) r= .22 for K = 297 independent correlations when sample sizes used in 
the 143 studies were taken into account and correlations were assumed to be sampled from 
a population of studies with varying SES-achievement correlations, and (c) T= .09 (K = 
297) assuming the sampled population of studies shared a common SES-achievement cor- 
relation). The pattern of modest correlations persisted across several variables believed to 
be related to SES-achievement correlations including type of SES measure (e.g., parental 
education, home possessions, FRL status), whether a school was public or private, and per- 
centage of Black, Hispanic, Asian, and White students in a study. 


Analyses of the relationship In sum, analyses of the relationship between 
between SES measures and SES measures and achievement show a wide 
achievement show a wide range ‘ange of correlations but generally agree that 
of correlations but generally average correlations are frequently less than 
agree that average correlations  -39; which is consistent with other empirical 


are frequently less than .30 summaries of this relationship*°“ but probably 
weaker than many educational researchers and 


policymakers expect. Modest SES-achievement correlations are consistent with reports in 
the social sciences that relationships among variables of all kinds are generally weak to mod- 
erately weak.#? 


Explanations of SES-achievement correlation patterns begin with ensuring that a broad dis- 
tribution of SES measures and their values have been sampled along with a range of student 
samples. Otherwise the distribution of SES measures and student samples may not reflect 
the U.S. as a whole and SES-achievement correlations may be biased. A review of coded vari- 
ables and findings in the above meta-analyses suggest a broad distribution of SES measures 
(parental education, occupation, income, home possessions, neighborhood SES, FRL status, 
composite measures) and values were sampled. The studies synthesized in the meta-analyses 
also represented a wide range of settings and conditions including school location (urban, 
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suburban, rural), type of achievement assessment (mathematics, literacy, science, GPA), 
source of SES information (parent, student, school), and student race, gender, and grade. 


Meta-analyses ideally reproduce results obtained from analyses of raw data,*? but various 
issues such as coding shortcomings can mask important patterns. An examination of the 
Coleman et al. (1966)*4 study (used in two of the above meta-analyses) provides evidence 
that meta-analytic inferences about the SES-achievement relationship can be made about a 
broad range of SES measures and student samples. Coleman et al.*5 examined the relation- 
ship between SES and achievement by sampling more than 600,000 students representing 
all 50 U.S. states. This study collected data in rural, suburban, and urban schools. Variables 
serving as SES measures included home possessions, mother’s and father’s education, and 
whether the father was working. Information on students’ verbal skill, reading comprehen- 
sion, and math achievement was also obtained. The resulting 936 correlations between SES 
and achievement measures produced values corresponding to the first, second, and third 
quartiles of the distribution of unweighted correlations of .06, .13, and .21, respectively. In 
other words, 25% of the 936 correlations were less than .06, one-half of the 936 correlations 
were less than .13, and 75% were less than .21, suggesting that SES and achievement were 
generally weakly related. Thus the results of Coleman et al. (1966)*° are consistent with me- 
ta-analytic evidence and suggest that inferences about the SES-achievement relationship 
can be made to a broad range of SES measures and student samples. 


One explanation for modest SES-achievement correlations is this pattern reflects the true 
SES-achievement relationship, implying that the impact of SES on achievement is not as 
strong as many educational researchers and policymakers assume. Alternatively this pattern 
of modest correlations may underestimate the true SES-achievement relationship, raising 
the question of what accounts for this pattern. Modest SES-achievement correlations cannot 
be solely attributed to an atheoretical approach to SES since the latter impacts definitions, 
interpretations, and selection of SES measures but would not be responsible by itself for a 
pattern of moderately weak to weak correlations. Nor is a pattern of modest correlations due 
to a failure to statistically capture the broader impact of SES by removing the effects of other 
variables influencing the SES-achievement relationship because these would be expected to 
be smaller than the original SES-achievement correlations. 


Measurement error in SES measures attenuates this relationship (lack of validity can also 
contribute to measurement error) but is unlikely to account for a pattern of modest correla- 
tions because the reliability of common SES measures like parental education is generally 
high.4” Characteristics of achievement assessments may play a role in the pattern of modest 
correlations but the necessary information to study their impact has not been reported. 
None of the above meta-analyses reported information about assessments which could pro- 
vide evidence that these correlations were attenuated due to the nature of the assessment 
(standardized tests, district- or teacher-constructed tests), number of items, and how items 
were scored (dichotomous, open-ended). 


Modest correlations between SES and educational outcomes are sometimes attributed to 
bias resulting from missing data because the nature of SES information makes nonresponse 
likely. As noted earlier, missing SES data can bias findings. For example, higher-income 
households show much higher rates of missing data than lower SES households, producing a 
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homogeneous sample of income values that reduces the resulting SES-achievement correla- 
tion. Similarly, middle school and high school students eligible for a FRL may not return 
application forms because of social stigma, which may have a homogenizing effect in that a 
disproportionate number of students who do not return forms are incorrectly classified as 
eligible for a FRL or not eligible for a FRL. This homogenization may reduce correlations. 
Smaller correlations may also be related to the reliance on the linearity of this relation- 
ship, which can produce a downward bias if the relationship differs at different values of 
SES (e.g., in the center of a distribution of SES values versus the tails). It is also likely SES 
measures moderate other effects in statistical models but these are not typically reported. 


There are fewer empirical summaries of the SES-achievement relationship for aggregated 
units like schools, although aggregated units are relatively common in educational research. 
White (1982)** reported r= .68 for K = 81 aggregated units (classrooms, schools, school 
districts) whereas Harwell et al. (2017)*9 found r= .20 for schools (K = 102). Individual 
studies of the relationship between school SES and achievement also show a wide range of 
correlations, for example .19 to .72.5° 


The effect of bias on measures can play an important role in these correlations.*' One source 
of bias arises when a school-level SES variable aggregated using student-level data such as 
FRL status is imperfectly correlated with the corresponding student-level FRL variable.” 
Another occurs when a school-level variable like percentage of FRL students is used as a 
predictor in statistical analyses along with student-level FRL in which case the former can 
be correlated with the residuals of the latter, biasing findings.*3 The size of aggregated units, 
for example schools versus districts, can also play a role, with smaller units showing less 
bias than larger units. The latter results can provide guidance when choosing among types 
of aggregated unit.%4 


The continued use of the profoundly flawed student FRL variable 


A third factor undermining the usefulness of common SES measures in educational research 
and policy applications is the continued practice of using student FRL to take disparities 
into account in statistical analyses. Students are certified as eligible for a reduced-price 
lunch if parental income is less than 185% of federal poverty guidelines or a free lunch if 
parental income is less than 130%. Available data for the 2013-2014 school year show that 
22 million students, approximately 44% of all K-12 students in the U.S., received a FRL. 
The Community Eligibility Provision is an extension of the FRL program that provides free 
breakfasts and lunches without a household application to all students in schools in low-in- 
come areas in which 40% or more of students are certified eligible for free school meals. 


The frequency with which student FRL is used as a measure of SES is non-negligible: the 
meta-analysis of Harwell et al. (2017)°° found 26% of the studies sampled fromi1980-2010 
used FRL (FRL was rarely used prior to 1980), and Sirin (2005)°*’ reported that 29% of sam- 
pled studies used FRL. FRL also appears regularly in studies using regional and national 
datasets, in large part because it is available for every student, usually accessible, and cheap 
to obtain. 
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Despite its frequent use, FRL is a profoundly flawed measure of SES whose deficiencies have 
been well-documented®* and include (a) its failure to accurately reflect financial resources, 
(b) reduced levels of participation in the FRL program in middle school and high school, 
especially 12" grade for reasons that remain unclear, although a social stigma may be partly 
responsible, and (c) an average misclassification rate of 20%.5° Issues with (a) and (b) are ex- 
acerbated when students who do not submit a FRL application have one submitted on their 
behalf by school officials (every student must have an FRL status), but the extent to which 
this occurs across school districts and states is unclear. Issues with (c) are exacerbated by 
uncertainty of the nature of the misclassification. A 20% misclassification rate in a school 
could mean one-half of the students eligible for FRL under federal guidelines are classified 
as ineligible and one-half who are ineligible are classified as eligible, or all 20% are misclas- 
sified as ineligible or eligible, or something in between. A misclassification rate of 20% could 
also be distributed equally or unequally across key student subgroups. There is no consistent 
evidence of the nature of misclassification in the FRL literature but misclassification of any 
kind undermines the validity of FRL status as an SES measure. 


Aggregating student FRL to the school level produces a variable frequently used as a predic- 
tor in statistical analyses. Perhaps more importantly, school FRL is widely used as an SES 
measure in policy applications where it is typically treated as a surrogate for school poverty. 
As noted earlier, eligibility for Title I funds depends on school FRL and schools with FRL 
rates equal to or greater than 40% are automatically eligible.°° Unsurprisingly the flaws of 
student FRL are transmitted to school FRL. If 20% of students in a school are misclassified, 
a school FRL rate of 40% would be distorted, although the precise nature of the distortion 
or its distribution across student subgroups would typically be unclear. In the absence of 
such distortion a school might or might not be eligible for Title I funding. Nor should school 
FRL be used to infer the FRL status of individual students because of the risk of biased in- 
ferences.® On the other hand, school or district FRL can provide a crude but useful index of 
economic need relative to other schools or districts. 


Recent Developments 


Fortunately, alternatives to commonly used SES measures have received attention. A U.S. 
Department of Education (2012)°? report advocated the use of composite SES indices when 
these are consistent with research or policy applications. Meta-analytic results show 38% of 
SES measures used in the sampled studies involved a composite measure, but the U.S. De- 
partment of Education report suggests greater consideration be given to composites. Other 
SES measures have appeared, including a student’s subjective SES, which has been shown 
to be a useful predictor of various outcomes,®? and a measure capturing how long a student 
has been economically disadvantaged as reflected in FRL eligibility. 


The U.S. Department of Education (2012)°5 report also recommended aggregating student 
variables such as the percentage of English Language Learners and average parental educa- 
tion level to the school level and using these as SES measures. Other authors have studied 
the use of census-based (aggregate) indicators of SES. Kurki et al. (2005)®° examined four 
aggregate measures of poverty which they argued quantify neighborhood disadvantage (dis- 
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similarity index, isolation index, poverty level of the school, and percentage of single parents 
in a school neighborhood), all of which require census data (e.g., block groups located within 
a one-kilometer radius around a school). A U.S. Department of Education (2015a)°*’ report 
described alternative measures of SES including the Census Bureau’s Small Area Income 
and Poverty Estimates (SAIPE), which provide school district poverty estimates to support 
the administration and allocation of Title I funding under the No Child Left Behind Act of 
2002. Still, school FRL remains a standard indicator of school poverty in law and in educa- 
tional practice, bolstered by the relative ease of access and low cost. 


Discussion and Analysis 


The prominent role of SES in educational research and policy is indisputable, as is a deeply 
rooted belief that SES has a pronounced impact on educational outcomes. However three 
factors can undermine the usefulness of common SES measures in ways that can bias re- 
search- and policy-based conclusions: (a) an atheoretical approach to conceptualizing SES 
and selecting measures which creates uncertainty about what SES represents; (b) modest 
correlations between SES measures and outcome variables that limit the ability of SES to 
take disparities between students, classrooms, and schools into account in statistical anal- 
yses, and (c) the continued use of the profoundly flawed student FRL variable as a measure 
of SES. These factors can produce biased conclusions but also provide a prescription for 
change centered on conceptualizing this construct and selecting measures in ways that clar- 
ify its interpretation and promote a deeper understanding and more effective use of SES. 


Uncertainty about what SES represents can be remedied by adopting a theoretical model 
that conceptualizes SES in ways consistent with the definition and purpose of the research 
or policy application and clarifies its interpretation. These models also provide guidance in 
selecting appropriate SES measures. For example, adopting a theoretical SES model focused 
on access to material capital and access to human capital would point to the use of paren- 
tal income as a measure of material capital and parental education as a measure of human 
capital. 


Unfortunately, selecting a theoretical model is challenging given the number of models in 
the social class literature, their complexity, and the range of available methodologies for 
measuring SES. What is needed is research unpacking these theories in ways that embed 
specific practices and measurement strategies in educational contexts, and in the process, 
adapts them to support research and policy applications. Work in health research®*®? pro- 
vides examples of how to proceed, which are characterized by a multidisciplinary approach 
involving expertise in a substantive area of education such as mathematics education, as 
well as psychometrics and statistical analysis, and social class. A multidisciplinary approach 
is also needed to develop new SES measures guided by theoretical SES models. 


The use of structural equation diagrams and regression-based (structural equation) mod- 
eling can assist in this process. In some instances the focus will be on providing evidence 
of construct validity for SES measures, but in others it will be on specifying and testing the 
impact of SES on educational outcomes. Figure 1 is an example of the latter and allows direct 
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effects, indirect effects, and the total effect of SES on an outcome to be estimated and test- 
ed. Central to constructing and testing an SES model is recognizing that SES-achievement 
correlations are modest on average but show a wide range of values, which makes it import- 
ant to select SES measures with non-negligible correlations with educational outcomes to 
enhance their effectiveness as control variables used to take into account disparities among 
students, classrooms, and schools in statistical analyses. 


Attention to theoretical SES models should also inform thinking about the mechanisms of 
influence linked to a wide range of SES values, about the limitations of SES measures in spe- 
cific contexts, and about the importance of differentiating the impact of SES on educational 
outcomes. Theoretical models may also prompt educational researchers and policymakers 
to consider alternative measures such as a student’s subjective SES. If an aggregate measure 
of SES is desired, student-level measures can be aggregated (e.g., percentage of English Lan- 
guage Learners in a school), or an aggregate measure like the Census Bureau’s Small Area 
Income and Poverty Estimate (SAIPE) may be appropriate. The net effect may be greater use 
of composite measures or aggregate measures like school-level percentage of English Lan- 
guage Learners and average parental education, and the development of new SES measures. 
In none of these scenarios is there a role for FRL status as a measure of SES at the student 
level, although school and district FRL will continue to be a crude but useful way to compare 
the economic need of different schools. 


Recommendations 


A theory-grounded model of SES should be adopted to define this construct in 
ways consistent with the purpose of the research or policy application. 


Correlations between SES measures and outcomes should be examined to assess 
the usefulness of these measures as control variables in statistical analyses. 


Researchers and policymakers wishing to employ existing SES measures should 
consider a composite index of SES, perhaps in conjunction with common mea- 
sures, or turn to alternative measures such as (a) students’ perception of their 
SES or (b) poverty estimates at the district level. Those interested in developing 
new measures should use a theory-grounded SES model as a guide to help ensure 
new SES measures do in fact measure what they are intended to (i.e., show evi- 
dence of construct validity). 


The development of new SES measures guided by a theory-grounded model of 
SES requires assembling a multidisciplinary team with expertise in a substantive 
area of education (e.g., mathematics education) as well as expertise in psycho- 
metrics, statistics, and the SES literature. 


Eligibility for a free- or reduced-price lunch should not be used as a student-level 
SES measure, but aggregating this variable to reflect the percentage of students 
receiving subsidized meals produces a crude but useful index to compare the eco- 
nomic need of a school or district with other schools or districts. 
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